fix(eval): agent cost was the wrong model's price, and the budget was the wrong unit - #545
Conversation
… the wrong unit Two defects found auditing OpenAI's 2026-07-30 GPT-5.6 price cut, plus the end-of-run tally that would have surfaced either one sooner. 1. PER-HARNESS PRICING. Executors that compute cost from token counts used their own hardcoded table, so the codex harness billed EVERY model it ran at gpt-5-codex's $1.25/$10 per 1M — gpt5-6-luna and gpt5-4-mini banked at identical rates despite really costing $0.20/$1.20 and $0.75/$4.50. Task.Pricing now carries the per-model rates and ResolveCostModel prefers them; a present-but-zero rate is honoured so free local models are not billed at cloud prices. The cost-KILL path was already per-model correct, which meant banked cost and kill threshold came from two different price tables: luna/graph_bfs banked $0.34259 at codex rates while the $0.30 budget that spared it saw $0.26980 at luna's. 2. COST PROVENANCE. cost_usd said nothing about whether anyone was charged. codex (auth_mode chatgpt) and claude (OAuth) run on subscriptions, so their dollars are list-price equivalents, not spend — yet they summed with metered OpenRouter/Vertex spend under one label, directly beneath a v1.0 KPI whose numerator is attributable METERED dollars. Each executor now classifies its own auth lane; banked as cost_provenance, stored on chain stages (schema v17), split out by ClassifyStageCost into a new `subscription` status that CostRollup keeps OUT of TotalKnownCost. Absent labels read as unknown, never metered — nothing is backfilled. 3. WORK GATES. A dollar ceiling buys tokens in inverse proportion to price, so nominally-uniform $0.30/$0.50 caps spanned 24x in real work: claude-sonnet-4-6, the longitudinal ANCHOR, got 0.14M tokens against deepseek-v4-flash's 3.40M — it had no budgets block at all and fell through to a formula that buys almost nothing at $3/$15. A suite asking "does the agent loop rescue weak models" gave the weakest the most rope. budgets.max_tokens_per_bench sets one shared 3.0M ceiling; subscription lanes gate on tokens alone and now sit identical. Task.MaxTokensPerBench had been plumbed since M-EVAL-OS-LONGITUDINAL but ONLY opencode honoured it; codex and claude now enforce it too. Metered opencode lanes keep their dollar ceiling as a real spend control with tokens as a second bound — still cost-limited, but for a stated reason. eval-suite now prints a cost tally at the end, split by provenance, so the metered figure is never conflated with subscription arithmetic again. Verified: build, go vet, make check-boundaries, and tests across executor, eval_harness, observatory, eval_analysis, coordinator, server and storage. Tally rendered against the real v0.30.0 baseline (2,090 rows, all correctly classified unknown — they predate the label). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The provenance and token-gate work pushed both executors past the 800-line check-file-sizes gate (codex 817, claude 843) — a CI failure, not a warning. Extracted into a cost.go per package: authLane() + CostModel() for both, plus getModel() for claude. These belong together — what a run costs and whether anyone was charged for it are the same question asked twice. codex 817 -> 761, claude 843 -> 778. No behaviour change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Mission-loop triage (iteration 150). This PR is an orphan: it has zero mentions in the V1 mission charter and zero in the mission log (control: Its purpose is still live — I checked rather than assuming. The standing rule here is that an
Control: So the two defects it fixes are still present: agent cost banked from the executor's hardcoded table rather than per-model rates, and State: Not actioned this iteration — a triage sweep never outranks the queue head by itself, and this is a rebase-and-revalidate job, not a merge. Filed as a queue row in the charter so it stops being invisible to the loop. 🤖 Generated with Claude Code |
… scope decision after 2 blocked quorum rounds
Pick: the queue head, m-net-effect-proxy-boundary (D5 Option B). NEW-DOC + quorum.
Designer codex gpt-5.6-sol (rotation advanced from claude:claude-fable-5).
Outcome: 662-line design doc with 19 verification rows and a silent-revert check on
every acceptance criterion. Quorum BLOCKED twice; item parked needs-human-review on
one decision (D-6). Planner/executor/evaluator not fired.
R1 caught a genuine design defect (gemini-3-1-pro): target-IP resolution specified in
TWO places — the existing preflight resolveAndValidateIP and the new RoundTripper — a
TOCTOU DNS-rebinding race plus a broken-proxied-request risk on hosts without external
DNS. Routed without a controller-invented resolution; the designer made the direct
RoundTripper the sole resolve-validate-dial site and skips local target DNS entirely on
proxy routes.
R1's other objection was an unverified premise, so the controller measured it rather
than forwarding it: production has 0 custom RoundTrippers, 0 DefaultTransport uses,
0 Transport.Clone, no shared HTTP factory (control: 29 inline http.Client{} sites).
That pass also produced the fact the doc lacked — Go's DefaultTransport sets
Proxy: ProxyFromEnvironment (transport.go:46-48, go1.26.5) — so bare clients are
already inside the egress boundary and only hand-built nil-Proxy transports can
escape. That turns "we found seven sites" into "seven is all there can be".
R2's surviving objection asks that the completeness gate become a go/packages AST
analyzer. Not carve-out-eligible (expands scope, needs judgment), so Standing rule 2
applies: park, do not force through. The decision was made cheap to answer by testing
the reviewer's own hypothesis — all five constructions the analyzer would catch are
ZERO at HEAD, each with a firing control. Two of those five controls failed on first
run and were re-run before any number was used.
Also: corrected the stale sweep-batch row (#588 was already closed 2026-08-05; and M2
gated the live-network SUBTEST at net_test.go:364, not the whole TestNetHttpPost
function). Filed orphaned PR #545 as a queue row — found by iteration 149's new
died-mid-flight check on its first independent use; 125 commits behind, zero charter
and zero log mentions, and its purpose is NOT superseded (measured at HEAD).
Retro — one skill edit (>=2-instance bar, purely additive: 38 insertions, 0 deletions):
Gate 2 gains rule 3f — a reviewer's objection is a claim too, and on an
"unverified premise" objection the controller's job is to MEASURE it, because the
measurement can refute the objection outright or shrink the work. Instance 1:
iteration 126. Instance 2: this iteration.
Metered: $0.1785 of the $5 ceiling (quorum x2 only; codex is OAuth-subscription).
Co-Authored-By: codex <gpt-5.6-sol>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR #545 sat open since 2026-07-31, 154 commits behind dev and CONFLICTING. The charter carried it as a "47 files ... rebase-and-revalidate job, not a merge". Measured rather than inherited: the merge produces exactly THREE conflicted files, one of them a changelog. Resolutions, each from reading both sides: - internal/eval_harness/agent_runner_multi.go — UNION. dev added a fallback that computes cost from tokens when the executor reports zero; this branch adds CostProvenance. They are complementary, and the fallback does NOT reintroduce the two-price-tables defect this PR exists to fix: CalculateCostWithBreakdown resolves through GlobalModelsConfig.Models[key], the same models.yml per-model rates the branch installs as task.Pricing. - cmd/ailang/eval_benchmark.go — took dev's side. dev extracted the agent path into cmd/ailang/eval_benchmark_agent.go, so the 427-line conflict is a code MOVE, not a rewrite. This branch's real delta to that block is four lines; two were re-applied verbatim into the extracted file (the CostProvenance field and the UpdateStageMetrics argument) and the third auto-merged in place. - changelogs/v0.18-current.md — union, [Unreleased] above [v0.33.1]. Zero content overlap between the two sides (control: 6 "Added" headings on dev's side, so the zero is a measurement). Validation, all outside any sandbox: - make test: rc=0, 107 packages ok, 0 FAIL (control: 7070 --- PASS lines, so the FAIL matcher would have fired). - go build ./...: the only failure is cmd/wasm, which fails IDENTICALLY on pristine dev — baselined rather than assumed. Zero new build failures. - make check-changelog / check-file-sizes / check-boundaries / fmt-check / vet: all rc=0. - models.yml lost nothing: 111 model keys on both sides, and every dev key is present in the merged file (comm -23 empty), including the pi-or-deepseek-v4-flash lane added after this branch was cut. - Migration numbering is clean: dev is at v16, this adds v17. - go.sum and go.mod are byte-identical before and after the run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…een, and the charter's own scale warning was a claim nobody had measured PICK: #545, because both items above it are parked on a human (#604 on D-2, #613 on D-1) and iteration 157's own Next names it as the fallback. HEADLINE: the charter has carried #545 since iteration 149 behind a warning reading "47 files ... a surface that moved a lot ... a rebase-and-revalidate job, NOT a merge". Every number in it is true and the conclusion is false. The merge produces THREE conflicted files, one hunk each: a changelog union with zero content overlap, a two-line struct-literal union, and one that looks enormous (427 lines ours vs 1 theirs) and is a code MOVE — dev extracted the agent path into cmd/ailang/eval_benchmark_agent.go, and #545's real delta to that block is four lines. Three iterations deferred the item on that sentence. A "surface moved a lot" warning measures how much OTHER work landed, never how much of it collides. Also recorded: the union resolution justified rather than assumed (dev's new zero-cost fallback resolves through the same models.yml key as #545's task.Pricing, so it does not reintroduce the two-price-table defect); the changelog claims a v0.33.1 release for which NO TAG EXISTS (caught in my own dashboard draft before publishing); Sonar at six consecutive analysed reds, measured on HEAD; and the Gate-0 weekly sweep, which found 7 zero-mention issues, closed #598 as a verified duplicate of #602, and batched the other six into one queue row positioned below every existing pick. STATUS rotation done with the arithmetic asserted before writing (before=after=1864, moved=1) and a post-edit queue-row grep, per the rules iteration 127's mass-deletion bought. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Unblocked by mission-control iteration 158 — The charter has deferred this PR for three iterations behind a warning that read: "47 files across Measured: 3 conflicted files out of 47, one hunk each.
The union in Validation, all outside any sandbox, all baselined:
Both defects this PR documents were re-verified as still live on Ready to merge on its own CI green. 🤖 Generated with Claude Code |
|
…d I was wrong that Sonar's red was purely inherited #545 merged as squash 32583be. Both gates green with completeness asserted: PR run present==4/4 all success, all four REQUIRED contexts pass; post-merge dev run present==3/3 all success. Queue row flipped to [LANDED] only on the observed green. Evaluator (sonnet, independent of the opus controller that resolved the merge): PASS 77/100, zero blocking. Both substantive findings reproduced first-party before acting. The cost-fallback/provenance union is correct for a reason the judge did not reach: ResolveCostProvenance is TOTAL, so provenance is a property of the auth lane rather than of who computed the number. Its coverage finding is real and is now #615, along with pi's hardcoded AuthLaneBilled. Refuted one of the judge's corrections by measurement: its 108 packages vs my 107 is scope, not error — make test excludes /scripts, of which exactly one package has tests. THE CORRECTION THAT MATTERS: earlier in this same iteration I recorded Sonar's red as "a standing condition inherited from earlier Go work, not a regression from this push". True of iterations 155-157, NOT true of this one. #545's head read 63.4% coverage on new code and the merged commit reads 73.7%, down from 78.7%. It never gated (non-required), so the merge decision stands, but "inherited" was an assumption carried across a push that changed the input — rule 3d exactly. Likewise the consecutive count: I wrote 7 by extrapolation and enumeration says 8. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>


Found while confirming OpenAI's 2026-07-30 GPT-5.6 price cut. Three defects, one new tally.
1. Agent cost billed the wrong model
Executors that compute cost from tokens used their own hardcoded table, so the codex harness billed every model it ran at gpt-5-codex's
$1.25/$10per 1M.gpt5-6-lunaandgpt5-4-minibanked at identical rates despite really costing$0.20/$1.20and$0.75/$4.50.Task.Pricingnow carries per-model rates;ResolveCostModelprefers them and falls back to the executor's table. A present-but-zero rate is honoured, so free local models aren't billed at cloud prices.The cost-kill path was already per-model correct — which meant banked cost and kill threshold came from two different price tables:
cost_usd2.
cost_usdnever said whether anyone was chargedcodex(auth_mode: chatgpt) andclaude(OAuth) run on subscriptions, so their dollars are list-price equivalents — yet they summed with metered OpenRouter/Vertex spend under one label, directly beneath a v1.0 KPI whose numerator is attributable metered dollars.Each executor now classifies its own auth lane. Banked as
cost_provenance, stored on chain stages (schema v17), and split byClassifyStageCostinto asubscriptionstatus thatCostRollupkeeps out ofTotalKnownCost. Absent labels read as unknown, never metered — nothing is backfilled by guess.3. The budget was denominated in the wrong unit
A dollar ceiling buys tokens in inverse proportion to price, so nominally-uniform
$0.30/$0.50caps spanned 24× in real work:claude-sonnet-4-6— the longitudinal anchor everything else is scored against — was the most starved model in the suite. It had nobudgets:block at all and fell through to a formula that buys almost nothing at$3/$15. A suite asking "does the agent loop rescue weak models" was giving the weakest the most rope.budgets.max_tokens_per_benchsets one shared 3.0M ceiling. Subscription lanes gate on tokens alone and now sit identical; their dollar caps were raised only so the token gate binds — not a spend increase, those lanes aren't billed. Meteredopencodelanes keep their dollar ceiling as a real spend control with tokens as a second bound.Task.MaxTokensPerBenchhad been plumbed toexecutor.Tasksince M-EVAL-OS-LONGITUDINAL, but onlyopencodeever honoured it.codexandclaudenow enforce it too.4. End-of-run cost tally
eval-suitenow prints per-model cost then totals split by provenance.METERED (actually billed)is the only line that answers "what did this cost us".Verification
Build,
go vet,make check-boundaries, and tests across executor, eval_harness, observatory, eval_analysis, coordinator, server and storage. Tally rendered against the real v0.30.0 baseline — 2,090 rows, all correctly classifiedunknown(they predate the label).Reviewer notes
$10.80,$3.60) are free on this rig's OAuth but become real per-benchmark exposure underAILANG_AUTH_MODE=apikeyin cloud dispatch. Flagged inline at both sites.cost_usd; correcting them would mean fabricating numbers, so the annotation route was taken (same call asCAVEATS.md).🤖 Generated with Claude Code